Image processing apparatus, image processing method, image processing program, and recording medium storing program

ABSTRACT

An image processing apparatus, an image processing method, a program, and a recording medium storing the program capable of extracting, from an input first image group, an image similar to an image extracted from a reference image group are provided. A first image group including a plurality of images of a user is transmitted to an order reception server. A second image group similar to the first image group is searched from a plurality of reference image groups stored in the order reception server. An image similar to an image previously extracted from the second image group is extracted from the first image group. An album is generated by arranging the image extracted from the first image group in a layout similar to a layout of an album generated from the second image group.

CROSS-REFRENCE TO RELATED APPLICATIONS

This application is a Divisional Application of U.S. Pat. ApplicationNo. 17/087,036 filed on Nov. 2, 2020, which is a Continuation of PCTInternational Application No. PCT/JP2019/017385 filed on Apr. 24, 2019,which claims priority under 35 U.S.C. § 119(a) to Japanese PatentApplications No. 2018-105976 filed on Jun. 1, 2018 and No. 2018-178638filed on Sep. 25, 2018. Each of the above application(s) is herebyexpressly incorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The invention relates to an image processing apparatus, an imageprocessing method, an image processing program, and a recording mediumstoring the program.

2. Description of the Related Art

In a case of generating an album from multiple images, images to be usedin the album have to be selected. However, it is difficult for a generaluser to know which image is to be selected. In the related art, forexample, it is considered that a reprint can be created in accordancewith requirements of a customer (JP2003-288350A), an image similar to adiagnosis target is searched from multiple images (JP2005-065728A), alayout template appropriate for preference of a user is created using atheme used in the past (JP2014-085814A), an image is searched using animage feature amount (JP2016-162963A), and a tomographic image isextracted (JP2017-127474A).

SUMMARY OF THE INVENTION

However, in the disclosure of JP2003-288350A, an image appropriate forthe reprint is found for one image. In the disclosure of JP2005-065728A,one image is found from multiple images. Thus, any of these disclosuresis not appropriate for extracting a part of images from an image groupincluding a plurality of images. In the disclosure of JP2014-085814A,the layout template is created. In the disclosure of JP2016-162963A, animage is searched using the image feature amount. In the disclosure ofJP2017-127474A, the tomographic image is extracted from a tomographicimage group. Thus, these disclosures are not appropriate for extractinga part of images from an image group including a plurality of images.

An object of the invention is to extract, from an input first imagegroup, an image similar to an image previously extracted from areference image group.

An image processing apparatus according to a first invention comprises areference image group reception device (reference image group receptionmeans) that receives an input of a plurality of reference image groups(instead of the reference image groups, may be feature amountsrepresenting a plurality of images included in the reference imagegroups), a first image extraction device (first image extraction means)that extracts a part of images from each of the plurality of referenceimage groups, a first image group reception device (first image groupreception means) that receives an input of a first image group (insteadof the first image group, may be feature amounts representing aplurality of images included in the first image group), an image groupsearch device (image group search means) that searches for a secondimage group (instead of the second image group, may be feature amountsrepresenting a plurality of images included in the second image group)similar to the first image group from the plurality of reference imagegroups, and a second image extraction device (second image extractionmeans) that extracts, from the first image group, an image similar tothe part of the images extracted from the second image group.

The first invention also provides an image processing method appropriatefor the image processing apparatus. That is, this method comprisesreceiving, by a reference image group reception device (reference imagegroup reception means), an input of a plurality of reference imagegroups, extracting, by a first image extraction device (first imageextraction means), a part of images from each of the plurality ofreference image groups, receiving, by a first image group receptiondevice (first image group reception means), an input of a first imagegroup, searching, by an image group search device (image group searchmeans), for a second image group similar to the first image group fromthe plurality of reference image groups, and extracting, by a secondimage extraction device (second image extraction means), from the firstimage group, an image similar to the part of the images extracted fromthe second image group.

The image processing apparatus according to the first invention maycomprise a processor that performs receiving an input of a plurality ofreference image groups, extracting a part of images from each of theplurality of reference image groups, receiving an input of a first imagegroup, searching for a second image group similar to the first imagegroup from the plurality of reference image groups, and extracting, fromthe first image group, an image similar to the extracted part of theimages.

The first invention also provides a computer-readable programcontrolling a computer of the image processing apparatus and a recordingmedium storing the program.

For example, extraction in the first image extraction device isprocessing used for arranging the extracted part of the images in analbum. In this case, the image processing apparatus may further comprisean image arrangement device (image arrangement means) that arranges, inthe album, the image extracted in the second image extraction device.

For example, the image arrangement device arranges, in the album, theimage extracted in the second image extraction device based oninformation about an album in which the part of the images extractedfrom the second image group is arranged.

For example, the information about the album is at least one of layoutinformation about the album or template information about the album.

For example, the image arrangement device arranges, in the album, theimage extracted in the second image extraction device by prioritizingmatching between a page order of the album and a time series of acapturing date and time of the image over arrangement of the image basedon the layout information about the album.

The image processing apparatus may further comprise a rearrangementdevice (rearrangement means) that obtains a middle capturing date andtime of capturing dates and times of images arranged on each page of thealbum and rearranges the pages of the album such that the middlecapturing date and time are in a time-series order.

The image processing apparatus may further comprise a first imagereplacement device (first image replacement means) that replaces animage arranged on one page with another image such that a capturing dateand time of the image arranged on one page of the album is earlier thanthe middle capturing date and time of capturing dates and times ofimages arranged on subsequent pages.

The image processing apparatus may further comprise a second imagereplacement device (second image replacement means) that replaces animage arranged on one page with another image such that a capturing dateand time of the image arranged on one page of the album is later thanthe middle capturing date and time of capturing dates and times ofimages arranged on previous pages.

The image processing apparatus may further comprise a third imagereplacement device (third image replacement) that replaces an imagearranged on one page with another image such that a capturing date andtime of the image arranged on one page of the album is earlier than aninitial capturing date and time of images arranged on subsequent pages.

The image processing apparatus may further comprise a fourth imagereplacement device (fourth image replacement means) that replaces animage arranged on one page with another image such that a capturing dateand time of the image arranged on one page of the album is later than alast capturing date and time of images arranged on previous pages.

For example, the image arrangement device arranges, in the album, theimage extracted in the second image extraction device by prioritizingmatching of a time series of a capturing date and time of the image overarrangement of the image based on layout information about the album forarrangement of the image in one page of the album.

For example, the image arrangement device arranges, in the album, theimage extracted in the second image extraction device by prioritizingarrangement of the image based on layout information about the albumover matching between a page order of the album and a time series of acapturing date and time of the image.

For example, the image arrangement device arranges, in the album, theimage extracted in the second image extraction device by prioritizingarrangement of the image based on layout information about the albumover matching of a time series of a capturing date and time of the imagefor arrangement of the image in one page of the album.

For example, the image group search device, in each reference imagegroup of the plurality of reference image groups, divides imagesincluded in the reference image group into a first small groupcollection of similar images, divides images included in the first imagegroup into a second small group collection of similar images, andsearches for a reference image group for which a degree of matchingbetween the first small group collection and the second small groupcollection is greater than or equal to a first threshold value as thesecond image group similar to the first image group.

For example, the image group search device combines small groups in acase where similarity between small groups included in the first smallgroup collection is greater than or equal to a second threshold value,and combines small groups in a case where similarity between smallgroups included in the second small group collection is greater than orequal to the second threshold value.

An image processing apparatus according to a second invention comprisesa first image group reception device (first image group reception means)that receives an input of a first image group, an image group searchdevice (image group search means) that searches for a second image groupsimilar to the first image group from a plurality of reference imagegroups each having information about an extracted part of images, and animage extraction device (image extraction means) that extracts, from thefirst image group, an image similar to the part of the images extractedfrom the second image group searched by the image group search device.

The second invention also provides an image processing methodappropriate for the image processing apparatus. That is, this methodcomprises receiving, by a first image group reception device (firstimage group reception means), an input of a first image group,searching, by an image group search device (image group search means),for a second image group similar to the first image group from aplurality of reference image groups each having information about anextracted part of images, and extracting, by an image extraction device(image extraction means), from the first image group, an image similarto the part of the images extracted from the second image group searchedby the image group search device.

Even in the second invention, the image processing apparatus maycomprise a processor that performs receiving an input of a first imagegroup, searching for a second image group similar to the first imagegroup from a plurality of reference image groups each having informationabout an extracted part of images, and extracting, from the first imagegroup, an image similar to the part of the images extracted from thesearched second image group.

The second invention also provides a computer-readable programcontrolling a computer of the image processing apparatus and a recordingmedium storing the program.

A third invention is an image processing system comprising an imageprocessing apparatus, and an image group search server, in which theimage processing apparatus includes a first image group reception device(first image group reception means) that receives an input of a firstimage group, the image group search server includes an image groupsearch device (image group search means) that searches for a secondimage group similar to the first image group from a plurality ofreference image groups each having information about an extracted partof images, and at least one of the image processing apparatus or theimage group search server includes an image extraction device (imageextraction means) that extracts, from the first image group, an imagesimilar to the part of the images extracted from the second image groupsearched by the image group search device.

An image processing apparatus according to a fourth invention comprisesa first image group reception device (first image group receptiondevice) that receives an input of a first image group, and an imageextraction device (image extraction means) that, among a plurality ofreference image groups each having information about an extracted partof images, extracts, from the first image group, an image similar to thepart of the images extracted from a second image group similar to thefirst image group.

The fourth invention also provides an image processing methodappropriate for the image processing apparatus. That is, this methodcomprises receiving, by a first image group reception device (firstimage group reception means), an input of a first image group, and amonga plurality of reference image groups each having information about anextracted part of images, extracting, by an image extraction device(image extraction means), from the first image group, an image similarto the part of the images extracted from a second image group similar tothe first image group.

Even in the fourth invention, the image processing apparatus maycomprise a processor that performs receiving an input of a first imagegroup, and among a plurality of reference image groups each havinginformation about an extracted part of images, and the image extractiondevice may extract, from the first image group, an image similar to thepart of the images extracted from a second image group similar to thefirst image group.

The fourth invention also provides a program controlling a computer ofthe image processing apparatus and a recording medium storing theprogram.

An image processing apparatus according to a fifth invention comprises afirst image group input device (first image group input means) thatreceives an input of a first image group, and an extraction device(extraction means) that extracts, from the first image group, aplurality of images for which similarity with a plurality ofconsultation images extracted from a consultation image group is greaterthan or equal to a threshold value.

The fifth invention also provides an image processing method appropriatefor the image processing apparatus. That is, this method comprisesreceiving, by a first image group input device (first image group inputmeans), an input of a first image group, and extracting, by anextraction device (extraction means), from the first image group, aplurality of images for which similarity with a plurality ofconsultation images extracted from a consultation image group is greaterthan or equal to a threshold value.

The image processing apparatus may comprise a processor that performsreceiving an input of a first image group, and extracting, from thefirst image group, a plurality of images for which similarity with aplurality of consultation images extracted from a consultation imagegroup is greater than or equal to a threshold value.

The fifth invention also provides a computer-readable programcontrolling a computer of the image processing apparatus and a recordingmedium storing the program.

A plurality of the consultation image groups may be present. The imageprocessing apparatus may further comprise a consultation image groupdetection device (consultation image group detection means) thatdetects, from the plurality of consultation image groups, theconsultation image group for which similarity with the first image groupis greater than or equal to a threshold value. The extraction device mayextract, from the first image group, a plurality of images for whichsimilarity with the plurality of consultation images extracted from theconsultation image group detected by the consultation image groupdetection device is greater than or equal to a threshold value.

Each consultation image of the plurality of consultation imagesextracted from the consultation image group may be pasted in an imagepasting region of a template. The image processing apparatus may furthercomprise an image pasting device (image pasting means) that pastes, inthe image pasting region of the template, an image for which similaritywith the consultation image pasted in the image pasting region of thetemplate is greater than or equal to a threshold value among theplurality of images extracted in the extraction device.

For example, the image pasting device pastes, in an image pasting regioncorresponding to the image pasting region in which the consultationimage is pasted, the image for which the similarity with theconsultation image pasted in the image pasting region of the template isgreater than or equal to the threshold value among the plurality ofimages extracted in the extraction device.

The image pasting device may paste, in an image pasting regioncorresponding to the image pasting region in which the consultationimage is pasted, an image for which the similarity with the consultationimage pasted in the image pasting region of the template is greater thanor equal to the threshold value and that is captured at a timing atwhich an image corresponding to the consultation image is expected to becaptured.

The image processing apparatus may further comprise a templatedesignation device (template designation means) that designates onetemplate from a plurality of templates. In this case, each consultationimage of the plurality of consultation images extracted from theconsultation image group may be pasted in the image pasting region ofthe template designated by the template designation device.

An order of the plurality of consultation images may be determined. Theimage processing apparatus may further comprise a first similarityadjustment device (first similarity adjustment means) that increases thesimilarity with the consultation image for an image captured in an ordercorresponding to the order of each consultation image of the pluralityof consultation images or an image captured at a timing at which animage corresponding to the consultation image is expected to be capturedamong images included in the first image group.

The image processing apparatus may further comprise a second similarityadjustment device (second similarity adjustment means) that increasesthe similarity with the consultation image for an image havinginformation similar to information about a face included in theconsultation image among images included in the first image group.

The image processing apparatus may further comprise a third similarityadjustment device (third similarity adjustment means) that increases thesimilarity with the consultation image for an image including a personfor which the number of appearances in the image is greater than orequal to a threshold value among images included in the first imagegroup.

The image processing apparatus may further comprise an image productcreation device (image product creation means) that creates an imageproduct using the images extracted by the extraction device.

The image processing apparatus may further comprise a person designationdevice (person designation means) that designates a desired person amongpersons appearing in images included in the first image group, and afourth similarity adjustment device (fourth similarity adjustment means)that increases the similarity with the consultation image for an imageincluding the person designated by the person designation device.

The image processing apparatus may further comprise a face imagedetection device (face image detection means) that detects a face imagefrom the images included in the first image group, and a face imagedisplay control device (face image display control means) that controlsa display device to display the face image detected by the face imagedetection device on a display screen. In this case, for example, theperson designation device may designate the desired person bydesignating the face image displayed on the display screen.

According to the first invention, the second image group similar to thefirst image group is searched from the plurality of reference imagegroups, and the image similar to the image extracted from the secondimage group is extracted from the first image group. Thus, an imagesimilar to an image previously extracted from the second image group isobtained from the first image group. Even in the second invention, thethird invention, and the fourth invention, an image similar to an imagepreviously extracted from the second image group is obtained from thefirst image group. According to the second invention, the same image asthe consultation image extracted from the consultation image group canbe obtained from the first image group.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a summary of an image processing system.

FIG. 2 is a block diagram illustrating an electric configuration of aclient computer.

FIG. 3 is a block diagram illustrating an electric configuration of anorder reception server.

FIG. 4 illustrates a layout information table and a template informationtable stored in an order reception database.

FIG. 5 is a flowchart illustrating a processing procedure of the imageprocessing system.

FIG. 6 is a flowchart illustrating a processing procedure of the imageprocessing system.

FIG. 7 is one example of image groups classified in a feature amountspace.

FIG. 8 illustrates a state where image groups are combined.

FIG. 9 is one example of image groups classified in the feature amountspace.

FIG. 10 is one example of image groups classified in the feature amountspace.

FIG. 11 illustrates a part of an album generated from images included ina second image group.

FIG. 12 illustrates a state where an image extracted from a first imagegroup is arranged.

FIG. 13 illustrates a part of an album generated from the imageextracted from the first image group.

FIG. 14 is a flowchart illustrating an image arrangement processingprocedure.

FIG. 15 illustrates a state where arrangement of images is changed.

FIG. 16 illustrates a state where arrangement of images is changed.

FIG. 17 illustrates a state where arrangement of images is changed.

FIG. 18 is a flowchart illustrating a processing procedure of an imageediting system.

FIG. 19 is a flowchart illustrating a processing procedure of the imageediting system.

FIG. 20 is a flowchart illustrating a processing procedure of the imageprocessing system.

FIG. 21 is a flowchart illustrating a processing procedure of the imageprocessing system.

FIG. 22 illustrates a state of a plurality of consultation image groupsstored in the order reception database.

FIG. 23 illustrates a content of an album created using a consultationimage extracted from a consultation image group.

FIG. 24 illustrates a first page of a consultation album.

FIG. 25 is one example of images included in the first image group.

FIG. 26 illustrates consultation images and images similar to theconsultation images.

FIG. 27 illustrates a state where an image extracted from the firstimage group is pasted in the consultation album.

FIG. 28 illustrates a first page of an album.

FIG. 29 is a part of a flowchart illustrating a processing procedure ofthe image processing system.

FIG. 30 is one example of a main person designation window.

FIG. 31 illustrates a flow and the like of wedding ceremony.

FIG. 32 illustrates a similarity calculation processing procedure.

DESCRIPTION OF THE PREFERRED EMBODIMENTS First Example

FIG. 1 illustrates a first example of the invention and illustrates asummary of an image processing system.

The image processing system is a system that extracts an image from aplurality of images and generates an album by laying out the extractedimage on a page.

An order reception server 5 (image group search server) and n (n is anatural number) number of client computers 1 to n are connected to theInternet. An order reception database 6 is connected to the orderreception server 5. Each of the client computers 1 to n and the orderreception server 5 can communicate with each other. Each of the clientcomputers 1 to n and the order reception server 5 constitute the imageprocessing system.

An album is generated by communication between any of the clientcomputers 1 to n and the order reception server 5.

FIG. 2 is a block diagram illustrating an electric configuration of theclient computer 1. Each of the client computers 2 to n has the sameconfiguration as the client computer 1.

An operation of the entire client computer 1 is managed by a centralprocessing unit (CPU) 10.

The client computer 1 includes a display device 11 that displays imagesand other information on a display screen, and a communication device 12that communicates with apparatuses and the like other than the clientcomputer 1 by connecting to the Internet and other networks. Inaddition, the client computer 1 includes a hard disk 13, a hard diskdrive 14 that accesses the hard disk 13, a memory 15 that stores dataand the like, and a keyboard 16 and a mouse 17 for inputting commandsand the like. Furthermore, the client computer 1 includes a compact discdrive 18 that accesses a compact disc 19, and a memory card reader andwriter 20 that writes data into a memory card 21 and reads out datarecorded on the memory card 21.

An operation program of the client computer 1 described later isreceived in the communication device 12 through the Internet. Thereceived operation program is installed on the client computer 1. Theoperation program may be recorded on a portable recording medium such asthe compact disc 19 and read out from the portable recording mediuminstead of being received by the client computer 1 through a networksuch as the Internet and installed on the client computer 1. In thiscase, the operation program read out from the portable recording mediumis installed on the client computer 1. Obviously, a computer (CPU 10) ofthe client computer 1 can read out the operation program.

FIG. 3 is a block diagram illustrating an electric configuration of theorder reception server 5 to which the order reception database 6 isconnected.

An operation of the entire order reception server 5 is managed by acentral processing unit (CPU) 30.

The order reception server 5 includes a communication device 31 thatcommunicates with apparatuses and the like other than the orderreception server 5 by connecting to the Internet and other networks. Inaddition, the order reception server 5 includes a hard disk 32, a harddisk drive 33 that accesses the hard disk 32, and a memory 34 thatstores data and the like. Furthermore, the order reception server 5includes a compact disc drive 35 that accesses a compact disc 36, and amemory card reader and writer 37 that writes data into a memory card 38and reads out data recorded on the memory card 38.

An operation program of the order reception server 5 described later isreceived in the communication device 31 through the Internet. Thereceived operation program is installed on the order reception server 5.The operation program may be recorded on a portable recording mediumsuch as the compact disc 36 and read out from the portable recordingmedium instead of being received by the order reception server 5 througha network such as the Internet and installed on the order receptionserver 5. In this case, the operation program read out from the portablerecording medium is installed on the order reception server 5.Obviously, a computer (CPU 30) of the order reception server 5 can readout the operation program.

FIG. 4 illustrates contents stored in the order reception database 6.

The order reception database 6 stores m (m is a natural number) numberof layout information tables and m number of template informationtables. The layout information table stores layout information for analbum generated in the past. The template information table storestemplate information used in the album generated in the past.

Table 1 is one example of one layout information table of the m numberof layout information tables.

TABLE 1 Image ID Feature Amount Selection Page Position Size 1 -- Y 1 0,0 60 × 40 2 -- N - - - 3 -- Y 1 100, 100 20 × 30 4 -- N - - - | | | | || | | | | | | | | | | | | 100 -- N - - -

The layout information table stores a feature amount of each of aplurality of images (referred to as a reference image group) incorrespondence with an image ID. The feature amount may be resolution ofthe image, a data amount, a degree of blurriness, a type of mainsubject, a relative size of the main subject with respect to the image,a position of the main subject, a tint, and the like. The feature amountmay also be generated by combining a plurality of above feature amounts.Furthermore, the feature amount may also be configured with a pluralityof parameters output by a learned model by receiving an input of theimage. The learned model is learned in advance by supervised learning orunsupervised learning. The feature amount output from the learned modelmay not be interpreted by a human being. However, at least a numericalvalue group that is uniquely output in a case where one image is inputcan be used as the feature amount according to an embodiment of thepresent invention. The feature amount is information necessary fordeciding appropriateness as an image (image extracted from the pluralityof images) to be used in the album, an arrangement position of thealbum, and the like. The layout information table also stores selectioninformation about a certain user as to whether an image is used or notused in the album among the plurality of images, a page of the usedimage in the album, and information about each of a position and a sizeof the image. In the selection information, assignment of “Y” indicatesan image used in the album, and assignment of “N” indicates an image notused in the album. The position is indicated by coordinates in a casewhere the upper left of each page of the album is set as an origin (0,0). The size is represented by (lateral length) mm × (longitudinallength) mm of the image.

For example, an image of an image ID 1 is selected as images to be usedin the album by a user of each of the client computers 1 to n based onthe selection information “Y”, is extracted by the CPU 10 (a part imageextraction device) or the like, is used in page 1 of the album based onpage information “1”, is positioned based on positional information (0,0) such that an upper left position of the image matches an upper leftposition of the page, and has a lateral length of 60 mm and alongitudinal length of 40 mm based on a size of 60 × 40. Similarly, itis perceived that an image of an image ID 2 is not used in the albumbased on the selection information “N”. As described above, the partimage extraction device according to the embodiment of the presentinvention can potentially extract a part of images from any referenceimage group. However, in extraction of an image from a first imagegroup, it is not necessary to actually extract a part of images from allrecorded reference image groups. The part image extraction deviceaccording to the embodiment of the present invention may simply extracta part of images from a second image group.

Table 2 is one example of one template information table of the m numberof template information tables.

TABLE 2 Page Template ID 1 3 2 7 3 5 | | | | | | 16 20

The template information table of Table 2 corresponds to the layoutinformation table of Table 1. In each layout information table, a layoutinformation table ID for identifying the layout information table isassigned. Also in each template information table, a templateinformation table ID for identifying the template information table isassigned. The template information table corresponding to the layoutinformation table is managed using the layout information table ID andthe template information table ID.

The layout information table and the template information table areinformation about an album of the reference image group. A plurality oflayout information tables and template information tables are stored inthe order reception database 6. Thus, information about an album of aplurality of reference image groups is stored in the order receptiondatabase 6. The order reception database 6 may store not only theinformation about the album of the plurality of reference image groupsbut also the plurality of reference image groups (image data) inaddition to the information about the album of the plurality ofreference image groups. In a case where the plurality of reference imagegroups are stored, the feature amount of the image is perceived byinterpreting the plurality of reference image groups. Thus, the layoutinformation table and the template information table may not be storedin the order reception database 6.

As will be described later, each time an order of the album is placed,the layout information table and the template information table aregenerated in the order reception server 5 by transmitting the layoutinformation and the template information to the order reception server 5from the client computer and receiving the layout information and thetemplate information (information about the album of the reference imagegroup) in the communication device 31 (reference image group inputdevice) of the order reception server 5. The generated layoutinformation table and template information table are stored in the orderreception database 6 by the order reception server 5.

FIG. 5 and FIG. 6 are flowcharts illustrating a processing procedure ofthe image processing system and illustrate processing procedures of theorder reception server 5 and any client computer of the client computers1 to n. In this example, communication is performed between the clientcomputer 1 and the order reception server 5 (in the first example, theorder reception server 5 is one example of an image processingapparatus).

The memory 15 of the client computer 1 stores image data representing aplurality of images (first image group) to be used in the album. Forexample, in a case where data representing the first image group isstored in the memory card 21, the data representing the first imagegroup is input into the client computer 1 from the memory card 21 by thememory card reader and writer 20. In a case where the client computer 1is a smartphone or the like, image data representing a plurality ofimages obtained by accumulation each time capturing is performed by acamera of the smartphone is the data representing the first image group.

The image data representing the first image group is transmitted to theorder reception server 5 from the client computer 1 (step S41 in FIG. 5). Instead of the image data representing the first image group, datarepresenting the feature amount of the image included in the first imagegroup may be transmitted to the order reception server 5 from the clientcomputer 1. The data representing the feature amount may be detected inthe client computer 1 or may be detected in other apparatuses other thanthe client computer 1.

In a case where the data representing the first image group (or datarepresenting a feature of the first image group) transmitted from theclient computer 1 is received in the communication device 31 (a firstimage group reception device) of the order reception server 5 (step S51in FIG. 5 ), information about an album of the second image groupsimilar to the first image group is searched by the CPU 30 (an imagegroup search device) of the order reception server 5 from theinformation about the album of the plurality of reference image groupsconsisting of the layout information table and the template informationtable stored in the order reception database 6 (step S52 in FIG. 5 ). Ina case where the plurality of reference image groups are stored in theorder reception database 6, the second image group may be searchedinstead of information about the second image group. In a case where thesecond image group is searched, the selection information foridentifying an image used in the album among a plurality of imagesincluded in the second image group is also read out.

Here, “similar image groups” mean that images included in two imagegroups are similar to each other. In addition, “searching for the secondimage group similar to the first image group” means that an “image groupincluding a plurality of images similar to the plurality of imagesincluded in the first image group is found as the second image groupfrom a plurality of image groups”.

The information about the album of the second image group similar to thefirst image group is searched as follows.

FIG. 7 illustrates a state where the plurality of images included in thefirst image group are classified into a plurality of groups by firststage classification.

In FIG. 7 , a horizontal axis denotes a first feature amount, and avertical axis denotes a second feature amount. While a state where theplurality of images are classified into a plurality of groups by twofeature amounts is illustrated, the number of feature amounts may not betwo, and three or more feature amounts may be used. In FIG. 7 , theplurality of images of a classification target are divided into 100groups of a group G1 to a group G100 by the CPU 30. For example, thisgroup division can be implemented using a k-means method. Images areincluded in each group of the group G1 to the group G100. Even in a casewhere actual images are not present, corresponding images (data foridentifying images) may be divided into groups from feature amounts ofthe images.

FIG. 8 illustrates a state of classification into a plurality of groupsof the group G1 to the group G100.

Next, the CPU 30 determines whether or not similarity between groups isless than a threshold value, and groups for which similarity is lessthan the threshold value are combined with each other. For example, acentroid in a feature amount space illustrated in FIG. 7 may be obtainedfor each group of the group G1 to the group G100, and similarity betweencentroids may be used as similarity between groups.

For example, in a case where similarity between the group G1 and thegroup G2 (similarity between a centroid of the group G1 and a centroidof the group G2) is less than the threshold value, a new group G101 isgenerated by combining the group G1 with the group G2. In a case wherethe group G1 and the group G2 are regarded as small groups, this casecorresponds to combining small groups in a case where similarity betweenthe small groups is greater than or equal to a threshold value (secondthreshold value). Similarly, in a case where similarity between thegroup G3 and the group G4 is less than the threshold value, a new groupG102 is generated by combining the group G3 with the group G4.Similarly, a new group G149 is generated by combining the group G97 withthe group G98, and a new group G150 is generated by combining the groupG99 with the group G100. Newly generated groups are also combined in acase where similarity between the groups is less than the thresholdvalue. For example, a new group G201 is generated by combining the groupG101 with the group G149.

In a case where groups for which similarity therebetween is less thanthe threshold value are not present anymore, the images included in thefirst image group are divided into a collection of small groups (secondsmall group collection) of similar images as illustrated in FIG. 9 .

FIG. 9 illustrates a feature amount space defined by the first featureamount on a horizontal axis and the second feature amount on a verticalaxis in the same manner as FIG. 7 .

In this feature amount space, the images included in the first imagegroup are divided into seven small groups of a group G301 to a groupG307. The CPU 30 calculates a centroid C301 to a centroid C307 in thegroups of the group G301 to the group G307, respectively.

Similarly, processing described with reference to FIG. 7 and FIG. 8 isperformed on feature amounts of the plurality of reference image groupsstored in the order reception database 6, and as illustrated in FIG. 10, each reference image group is divided into a collection of smallgroups (first small group collection) of images for which featureamounts of images included in the reference image group are similar.

FIG. 10 illustrates a feature amount space defined by the first featureamount on a horizontal axis and the second feature amount on a verticalaxis in the same manner as FIG. 9 .

In this feature amount space, the feature amounts of the images includedin the reference image group are divided into eight small groups ofgroups G401 to G408. A centroid C401 to a centroid C408 is alsocalculated in the group G401 to the group G408.

A distribution of the feature amounts of the images illustrated in FIG.10 is generated for each reference image group of the plurality ofreference image groups stored in the order reception database 6.

A sum of distances (degree of matching between the first small group andthe second small group) in the feature amount space between,respectively, the centroid C301 to the centroid C307 of the group G301to the group G307 generated for the first image group as illustrated inFIG. 9 and the centroid C401 to the centroid C408 of the group G401 tothe group G408 generated for the reference image group as illustrated inFIG. 10 is calculated for all reference image groups of the plurality ofreference image groups. A reference image group for which the sum ofdistances is less than or equal to a threshold value (corresponds to acase where the degree of matching between the first small group and thesecond small group is greater than or equal to a threshold value) issearched by the CPU 30 (an image group search device) as the secondimage group similar to the first image group.

In a case where the second image group is searched from the plurality ofreference image groups, a part of images extracted from the searchedsecond image group is found by the CPU 30, and an image similar to theextracted part of the images is extracted from the images included inthe first image group by the CPU 30 (a similar image extraction device)(step S53 in FIG. 5 ). The image extracted from the second image groupis perceived by referring to the selection information stored in thelayout information table corresponding to the second image group in aset of the layout information table and the template information tablestored in the order reception database 6. As described above, the imagefor which the selection information is “Y” is the extracted image.

FIG. 11 illustrates a part of pages of the album generated using theimage extracted from the second image group.

A template 70 is used in the part of the pages of the album. In thetemplate 70, regions 71, 72, and 73 in which images are pasted aredefined at the upper left, the lower left, and the right. Images I11,I12, and I13 are pasted in the regions 71, 72, and 73. Information(information such as which image is pasted in which region) about thepage 70 is perceived from the layout information table and the templateinformation table corresponding to the second image group.

The image extracted from the first image group is arranged in the albumby the CPU 30 (step S54 in FIG. 5 ). The image is arranged in the albumbased on album information included in the template information tableand the layout information table corresponding to the second image groupsimilar to the first image group.

FIG. 12 illustrates a state where the image extracted from the firstimage group is arranged in the album.

Images I21, I22, and I23 are extracted from the first image group asimages similar to the images I11, I12, and I13, respectively, extractedfrom the second image group. The same template as the template 70 inwhich the images I11, I12, and I13 extracted from the second image groupare pasted is found.

The image I21 of the first image group similar to the image I11 of thesecond image group is arranged in the region 71 in which the image I11is arranged. The image I22 of the first image group similar to the imageI12 of the second image group is arranged in the region 72 in which theimage I12 is arranged. Similarly, the image I23 of the first image groupsimilar to the image I13 of the second image group is arranged in theregion 73 in which the image I13 is arranged.

FIG. 13 illustrates a part of pages of the album generated using theimage extracted from the first image group.

The part of the pages of the album illustrated in FIG. 13 is generatedas the same album by consulting the part of the pages of the albumgenerated using the image extracted from the second image groupillustrated in FIG. 11 .

As is perceived by comparing FIG. 11 with FIG. 13 , similar imagesconstitute the albums in the same layout.

The template of the album is decided based on a template ID stored inthe template information table corresponding to the second image group,and the image extracted from the first image group is arranged in thealbum in accordance with a page, a position, and a size in which theimage extracted from the second image group is pasted. Accordingly, analbum similar to the album generated using the second image groupsimilar to the first image group can be automatically generated from thefirst image group.

In a case where the album is generated, processing such as correction ofarrangement locations of the images arranged in the generated album orchanging of the images is performed. Details of this processing will bedescribed later.

In a case where the album is generated from the first image group, thelayout information table and the template information table for thefirst image group are generated, and the generated layout informationtable and template information table are stored in the order receptiondatabase 6 by the CPU 30 of the order reception server 5 (step S55 inFIG. 5 ).

Next, album data that represents the album is transmitted to the clientcomputer 1 from the order reception server 5 (step S56 in FIG. 6 ).

In a case where the album data is received in the client computer 1(step S42 in FIG. 6 ), the album generated in the order reception server5 is displayed on the display screen of the display device 11 of theclient computer 1 (step S43 in FIG. 6 ).

The user of the client computer 1 checks the displayed album andcorrects the album in a case where the user thinks that the album needsto be corrected. For example, replacement of the image arranged in thealbum, changing of the arrangement position of the image, changing ofthe size of the image, or changing of the template is performed asnecessary. In a case where the album is corrected (YES in step S44 inFIG. 6 ), correction data that represents a corrected content istransmitted to the order reception server 5 from the client computer 1(step S45 in FIG. 6 ). Instead of transmitting the correction data, thealbum data after correction may be transmitted to the order receptionserver 5.

In a case where the correction data transmitted from the client computer1 is received in the order reception server 5 (YES in step S57 in FIG. 6), contents of the layout information table and the template informationtable of the second image group are updated based on the receivedcorrection data (step S58 in FIG. 6 ).

In a case where the user of the client computer 1 pushes an order button(YES in step S46 in FIG. 6 ), an order command is transmitted to theorder reception server 5 from the client computer 1 (step S47 in FIG. 6).

In a case where the order command transmitted from the client computer 1is received in the order reception server 5 (YES in step S59 in FIG. 6), order reception processing is performed. The album is generated usingthe layout information table and the template information tablecorresponding to the second image group among the layout informationtables and the template information tables stored in the order receptiondatabase 6. In a case where the album data is stored in the orderreception server 5, the album may be generated using the album data.

FIG. 14 is a flowchart illustrating an image arrangement processingprocedure (processing procedure of step S54 in FIG. 5 ). The processingprocedure illustrated in FIG. 14 sets capturing dates and times of theimages arranged in the generated album in a time-series order.

As described with reference to FIG. 11 to FIG. 13 , the images includedin the first image group are arranged (temporarily arranged) similarlyto the album of the second image group searched from the plurality ofreference image groups (step S81).

FIG. 15 illustrates a part of the album generated from the imagesincluded in the first image group.

A page image P1 of page 1, a page image P2 of page 2, and a page imageP3 of page 3 constituting a part of the album are represented on a leftside of FIG. 15 . An image 131, an image I32, and an image I33 arearranged at the upper left, the lower left, and the right of the pageimage P1, respectively. The image I31 is captured at 10:53 on May 13,2018. The image I32 is captured at 11:34 on May 12, 2018. The image I33is captured at 13:02 on May 13, 2018. An image I34, an image I35, and animage I36 are arranged at the left, the upper right, and the lower rightof the page image P2, respectively. The image I34 is captured at 7:33 onMay 12, 2018. The image I35 is captured at 10:28 on May 16, 2018. Theimage I36 is captured at 12:38 on May 16, 2018. An image I37, an imageI38, and an image I39 are arranged at the upper left, the lower left,and the right of the page image P3, respectively. The image I37 iscaptured at 13:04 on May 13, 2018. The image I38 is captured at 12:26 onMay 14, 2018. The image I39 is captured at 16:57 on May 16, 2018. Inthis stage, an order of arrangement of the images in the album may notnecessarily match an order of capturing dates and times of the images.Thus, the following procedure is subsequently performed in a case wherethe user desires.

First, a middle capturing date and time (middle capturing date and timemay be any of a middle value, an average value, a center value, or thelike of capturing dates and times of images arranged in each page) amongthe capturing dates and times of the images arranged in each page areread out. Among the image I31, the image I32, and the image I33 includedin the page image P1 of page 1, the image I31 is captured at 10:53 onMay 13, 2018. The image I32 is captured at 11:34 on May 12, 2018. Theimage I33 is captured at 13:02 on May 13, 2018. Thus, the middlecapturing date and time are 10:53 on May 13, 2018 of the image 131, andthis capturing date and time are read out (obtained) by the CPU 30 (stepS82). Similarly, among the image I34, the image I35, and the image I36included in the page image P2 of page 2, the image I34 is captured at7:33 on May 12, 2018. The image I35 is captured at 10:28 on May 16,2018. The image I36 is captured at 12:38 on May 16, 2018. Thus, themiddle capturing date and time are 10:28 on May 16, 2018 of the imageI35, and this capturing date and time are read out by the CPU 30.Furthermore, among the image I37, the image I38, and the image I39included in the page image P3 of page 3, the image I37 is captured at13:04 on May 13, 2018. The image I38 is captured at 12:26 on May 14,2018. The image I39 is captured at 16:57 on May 16, 2018. Thus, themiddle capturing date and time are 12:26 on May 14, 2018 of the imageI38, and this capturing date and time are read out by the CPU 30.

As illustrated on a right side of FIG. 15 , the page images P1, P2, andP3 are rearranged by the CPU 30 (a rearrangement device) such that themiddle capturing date and time read out are in a time-series order (stepS83). Page 1 is not rearranged. Page 2 is rearranged from the page imageP2 to P3. Page 3 is rearranged from the page image P3 to P2. Since atime-series order is set in a case of a view in units of pages, imagescan be arranged in a natural order as an album. However, in this stage,there is a possibility that a capturing date and time of a certainphotograph of a certain page are later than the middle capturing dateand time of a photograph group of the subsequent page. Therefore, thefollowing procedure is subsequently performed in a case where the userdesires.

Next, images arranged on one page are replaced with other images by theCPU 30 (a first image replacement device) such that capturing dates andtimes of the images arranged on one page of the album are earlier thanthe middle capturing date and time of capturing dates and times ofimages arranged on the subsequent page (step S84).

As represented on the right side of FIG. 15 , the image 131 included inthe page image P1 set as page 1 is captured at 10:53 on May 13, 2018.The image I32 is captured at 11:34 on May 12, 2018. The image I33 iscaptured at 13:02 on May 13, 2018. All of the images I31, I32, and I33are earlier than 12:26 on May 14, 2018 which is the middle capturingdate and time of capturing dates and times of images included in thepage image P3 of page 2 which is the subsequent page. Thus, the imagesI31, I32, and I33 included in the page image P1 of page 1 are notreplaced.

The image I37 included in the page image P3 set as page 2 is captured at13:04 on May 13, 2018. The image I38 is captured at 12:26 on May 14,2018. The image I39 is captured at 16:57 on May 16, 2018. The image I39having a capturing date and time later than 10:28 on May 16, 2018 whichis the middle capturing date and time of capturing dates and times ofimages included in the page image P2 of page 3 which is the subsequentpage is included. Therefore, the image I39 is replaced with an imagethat is similar to the image I39 and has a capturing date and timeearlier than 10:28 on May 16, 2018 which is the middle capturing dateand time of the capturing dates and times of the images included in thepage image P2 of the subsequent page.

Furthermore, images arranged on one page are replaced with other imagesby the CPU 30 (a second image replacement device) such that capturingdates and times of the images arranged on one page of the album arelater than the middle capturing date and time of capturing dates andtimes of images arranged on the previous page (step S85).

With reference to the page images P1, P2, and P3 represented on theright side of FIG. 15 , the image I37 included in the page image P3 (onepage) set as page 2 is captured at 13:04 on May 13, 2018. The image I38is captured at 12:26 on May 14, 2018. The image I39 is captured at 16:57on May 16, 2018. The images I37, I38, and I39 are later than 13:02 onMay 13, 2018 which is the middle capturing date and time of the imagesI31, I32, and I33 included in the page image P1 of page 1 which is theprevious page. Thus, the images of the page image P3 set as page 2 arenot replaced.

The image I34 included in the page image P2 (one page) set as page 3 iscaptured at 7:33 on May 12, 2018. The image I35 is captured at 10:28 onMay 16, 2018. The image I36 is captured at 12:38 on May 16, 2018. Theimage I34 having a capturing date and time earlier than 12:36 on May 14,2018 which is the middle capturing date and time of the images I37, I38,and I39 included in the page image P3 of page 2 which is the previouspage is included. Therefore, the image I34 is replaced with an imagethat is similar to the image I34 and has a capturing date and time laterthan 12:36 on May 14, 2018 which is the middle capturing date and timeof the capturing dates and times of the images included in the pageimage P3 of the previous page.

The page image P1 of page 1, the page image P3 of page 2, and the pageimage P2 of page 3 are represented on a left side of FIG. 16 . Theimages I31 to I39 included in the page images P1, P3, and P2 representedon the left side of FIG. 16 and arrangement of the images 131 to I39 arethe same as the images I31 to I39 included in the page images P1, P3,and P2 represented on the right side of FIG. 15 and arrangement of theimages I31 to I39. However, the images I39 and I34 to be replaced arehatched in order for the user to easily perceive the images I39 and I34.Instead of hatching, the images I39 and I34 to be replaced may be set tobe distinguishable from other images by making the images I39 and I34darker or brighter than the other images. The images I39 and I34 to bereplaced may not necessarily be set to be distinguishable from the otherimages.

As represented on a right side of FIG. 16 , by executing step S84 andstep S85, the image I39 included in the page image P3 of page 2 isreplaced with an image I40 having a capturing date and time of 16:57 onMay 14, 2018. The image I34 included in the page image P2 of page 3 isreplaced with an image I41 having a capturing date and time of 14:33 onMay 14, 2018. Accordingly, capturing dates and times of images arrangedon one page of the album are set to be later than the middle capturingdate and time of capturing dates and times of images arranged on theprevious page. In addition, the capturing dates and times of the imagesarranged on one page of the album are set to be earlier than the middlecapturing date and time of capturing dates and times of images arrangedon the subsequent page. However, therefore, in this stage, there is apossibility that a certain image of a certain page is later than aninitial capturing date and time of the images arranged on the subsequentpage. Therefore, the following steps are subsequently executed in a casewhere the user desires.

Images arranged on one page are replaced with other images by the CPU 30(a third image replacement device) such that capturing dates and timesof the images arranged on one page of the album are earlier than theinitial capturing date and time of images arranged on the subsequentpage (step S86).

As represented on the right side of FIG. 16 , the image I31 included inthe page image P1 (one page) set as page 1 is captured at 10:53 on May13, 2018. The image I32 is captured at 11:34 on May 12, 2018. The imageI33 is captured at 13:02 on May 13, 2018. The images 131, I32, and I33are captured earlier than 13:04 on May 13, 2018 which is the initialcapturing date and time of the images I37, I38, and I39 arranged in thepage image P3 of page 2 which is the subsequent page. Thus, the imagesI31, I32, and I33 included in the page image P1 are not replaced.

The image I37 included in the page image P3 (one page) set as page 2 iscaptured at 13:04 on May 13, 2018. The image I38 is captured at 12:26 onMay 14, 2018. The image I40 is captured at 16:57 on May 14, 2018. Thecapturing date and time of the image I40 are later than 14:33 on May 14,2018 which is the initial capturing date and time of the images I41,I35, and I36 arranged in the page image P2 which is the subsequent page.Thus, the image I40 included in the page image P3 is replaced with animage that is similar to the image I40 (or the image I39) and iscaptured at a capturing date and time later than 14:33 on May 14, 2018which is the initial capturing date and time of the images I41, I35, andI36 arranged in the page image P2 which is the subsequent page.

Images I31 to I33, I35 to I37, I41, and I42 included in the page imagesP1, P3, and P2 on a left side of FIG. 17 are the same as the images I31to I33, I35 to I38, I40, and I41 included in the page images P1, P3, andP2 displayed on the right side of FIG. 16 , and arrangement thereof isalso the same. However, the image I40 to be replaced is hatched in orderfor the user to easily perceive the image I40.

As represented on a right side of FIG. 17 , the image I40 of the pageimage P3 is replaced with the image I42 that is similar to the image I40and is captured at a capturing date and time later than 14:33 on May 14,2018 which is the initial capturing date and time of the images I41,I35, and I36 arranged in the page image P2 which is the subsequent page.The image I42 has a capturing date and time of 13:57 on May 14, 2018 andis captured at a capturing date and time later than 14:33 on May 14,2018 which is the initial capturing date and time of the images I41,135, and 136 arranged in the page image P2.

Next, images arranged in one page are replaced with other images by theCPU 30 (a fourth image replacement device) such that capturing dates andtimes of the images arranged on one page of the album are later than alast capturing date and time of images arranged on the previous page(step S87).

As represented on the right side of FIG. 17 , the image I37 included inthe page image P3 (one page) set as page 2 is captured at 13:04 on May13, 2018. The image I38 is captured at 12:26 on May 14, 2018. The imageI42 is captured at 13:57 on May 14, 2018. The images I37, I38, and I42are captured later than 13:02 on May 13, 2018 which is the lastcapturing date and time of the images I31, I32, and I33 arranged in thepage image P1 of page 1 which is the previous page. Thus, the imagesI37, I38, and I42 included in the page image P3 are not replaced.

The image I41 included in the page image P2 (one page) set as page 3 iscaptured at 14:33 on May 14, 2018. The image I35 is captured at 10:28 onMay 16, 2018. The image I36 is captured at 12:38 on May 16, 2018. Theimages I41, I35, and I36 are captured later than 13:57 on May 14, 2018which is the last capturing date and time of the images I37, I38, andI42 arranged in the page image P3 which is the previous page. Thus, theimages I41, I35, and I36 included in the page image P2 are also notreplaced.

By executing steps up to step S87, the capturing dates and times of theimages arranged in the album can be set in a time-series order. That is,extracted images can be arranged by prioritizing matching between a pageorder of the album and a time series of the capturing dates and times ofthe images over arrangement of the images based on layout informationabout the album.

In the above processing, processing of step S84 and processing of stepS85 are separately performed. However, images arranged on one page maybe replaced as one processing such that capturing dates and times ofimages arranged on one page of the album are later than the middlecapturing date and time of capturing dates and times of images arrangedon the previous page, and are earlier than the middle capturing date andtime of capturing dates and times of images arranged on the subsequentpage of one page.

In addition, instead of separately performing processing of step S86 andprocessing of step S87, images arranged on one page may be replaced asone processing such that capturing dates and times of images arranged onone page of the album are later than the last capturing date and time ofimages arranged on the previous page, and are earlier than the initialcapturing date and time of images arranged on the subsequent page of onepage.

Furthermore, while images are not arranged in accordance with the timeseries of the capturing dates and times of the images in one page of thealbum in the above example, the images extracted from the second imagegroup may be arranged by prioritizing matching of the time series of thecapturing dates and times of the images even in one page of the album.In this case, positions of regions formed in pages for pasting imagesand a time-series order of capturing dates and times are predeterminedfor each page, and the extracted images are arranged in accordance withthe determined order.

Furthermore, in the above example, the images extracted from the secondimage group may be arranged in the album by prioritizing arrangement ofthe images based on the layout information about the album over matchingbetween the page order of the album and the time series of the capturingdates and times of the images. In this case, arrangement of the imagesin the album may be finished in a state of temporary arrangement asdescribed above, and processing described with reference to FIG. 14 toFIG. 17 may not be performed.

In addition, in arrangement of images in one page of the album,processing described with reference to FIG. 14 to FIG. 17 may beperformed in a case where the images extracted in the second image groupare arranged in the album by prioritizing arrangement of the imagesbased on the layout information about the album over matching of thetime series of the capturing dates and times of the images.

According to the first example, an album can be generated using a layoutappropriate for images of the user among layouts in the past.

Second Example

FIG. 18 and FIG. 19 are flowcharts illustrating a processing procedureof the image processing system and illustrate processing procedures ofthe order reception server 5 and any client computer of the clientcomputers 1 to n. Even in the second example, communication is performedbetween the client computer 1 and the order reception server 5 (in thesecond example, the client computer 1 is one example of the imageprocessing apparatus).

Arrangement of images in the album (processing of step S14 in FIG. 5 )in the order reception server 5 in the first example is performed in theclient computer 1 in the second example.

As described above, the order reception database 6 connected to theorder reception server 5 stores m number of layout information tablesand m number of template information tables representing feature amountsof a plurality of reference image groups. The feature amounts of theplurality of reference image groups are transmitted to the clientcomputer 1 from the order reception server 5 (step S91 in FIG. 18 ).

In a case where the feature amounts of the plurality of reference imagegroups transmitted from the order reception server 5 are received in thecommunication device 12 of the client computer 1 (step S101 in FIG. 18), the received feature amounts are temporarily stored in the memory 15.

The memory card 21 stores the first image group including a plurality ofimages to be used for generating an album. The first image group storedin the memory card 21 is read out by the memory card reader and writer20 (a first image group reception device).

Feature amounts of the second image group similar to the first imagegroup read out are searched by the CPU 10 (an image group search device)from the feature amounts of the plurality of reference image groups(step S102 in FIG. 18 ). In processing of step S102 in FIG. 18 , thesame processing as processing of step S52 in FIG. 5 is performed in theclient computer 1. The second image group is substantially searched fromthe plurality of reference image groups. The feature amounts of theplurality of reference image groups include the layout informationtable. The layout information table includes the selection informationindicating whether or not any of the reference image groups is selectedas images to be used in the album. Thus, it can be said that thereference image groups have information about a part of images extractedfrom the reference image groups.

Next, images (for example, the images I21, I22, and I23 in FIG. 12 )that are similar to a part of images (for example, the images I11, I12,and I13 in FIG. 11 ) extracted from the second image group are extractedfrom the first image group by the CPU 10 (a similar image extractiondevice) of the client computer 1 (step S103 in FIG. 18 ). In processingof step S103 in FIG. 18 , the same processing as processing of step S53in FIG. 5 as described above is performed in the client computer 1.

The images extracted from the first image group are arranged in thealbum as illustrated in FIG. 13 using the same layout as a layout of thesecond image group (step S104 in FIG. 18 ). In processing of step S104in FIG. 18 , the same processing as processing of step S54 in FIG. 5 asdescribed above is performed in the client computer 1.

A content of the album in which the images extracted from the firstimage group are arranged is displayed on the display screen of thedisplay device 11 of the client computer 1. The user checks the content.As a result of checking, in a case where the content of the album needsto be corrected (YES in step S105 in FIG. 19 ), the user performscorrection (step S106 in FIG. 19 ).

In a case where the user of the client computer 1 pushes the orderbutton (YES in step S107 in FIG. 19 ), the order command and the albumdata are transmitted to the order reception server 5 from the clientcomputer 1 (step S108 in FIG. 19 ). As described above, the album datais data necessary for generating the album. The album in which theimages extracted from the first image group are arranged on each pagecan be generated using the album data.

In a case where the order command and the album data transmitted fromthe client computer 1 are received in the order reception server 5 (YESin step S92 in FIG. 19 ), the received album data is analyzed by the CPU30 of the order reception server 5, and the layout information and thetemplate information are obtained. The layout information table and thetemplate information table are newly generated from the obtained layoutinformation and template information and are stored in the orderreception database 6 (step S93 in FIG. 19 ). The order receptionprocessing is performed by generating the album from the received albumdata (step S94 in FIG. 19 ).

In the second example, the album can be generated in the client computer1 using a layout appropriate for images of the user among layouts in thepast.

In the second example, the first image group is input in the clientcomputer 1 (image processing apparatus) (client computer 1 comprises afirst image group input device). The second image group similar to thefirst image group is searched from the plurality of reference imagegroups in the order reception server 5 (image group search server).Images similar to a part of the images extracted from the second imagegroup are extracted from the first image group by the CPU 10 (Thesimilar image extraction device). However, the images similar to thepart of the images extracted from the second image group may beextracted from the first image group in the order reception server 5. Inthis case, identification data of the second image group is transmittedto the order reception server 5, and the images similar to the part ofthe images extracted from the second image group are extracted from thefirst image group by the CPU 30 (The similar image extraction device) ofthe order reception server 5 based on information stored in the layoutinformation table and the template information table.

Third Example

Even in the third example, the image processing system in which theplurality of client computers 1 to n and the order reception server 5communicate through the Internet is used in the same manner as the firstexample. In the third example, an example of creating an album of awedding ceremony will be described. However, an album of an event or thelike other than the wedding ceremony may also be created. Instead of analbum, for example, application for pasting a plurality of images on onepaper sheet can also be made. Any application can be made as long as aplurality of images are extracted from an image group (first image groupPg).

FIG. 20 and FIG. 21 are flowcharts illustrating a processing procedureof the image processing system and illustrate processing procedures ofthe order reception server 5 and the client computer 1. Any clientcomputer other than the client computer 1 among the client computers 1to n can also be used. In this example, communication is performedbetween the client computer 1 and the order reception server 5 (in thethird example, the order reception server 5 is one example of the imageprocessing apparatus). In FIG. 21 , the same processing as processingillustrated in FIG. 6 will be designated by the same reference sign, anddescription thereof will be omitted.

A plurality of image folders are formed in the memory card 21 (may beother recording media), and a plurality of images are stored in each ofthe plurality of image folders. An image folder in which images to beused in the album are stored is designated by the user. The user may bea bridegroom or a bride having a wedding ceremony or may be a businessperson such as an album creator. The plurality of images stored in thedesignated image folder are input from the memory card reader and writer20 (one example of the first image group input device) as the firstimage group Pg (step S110) and are read out by the client computer 1.Data representing the plurality of images included in the first imagegroup Pg is transmitted (transmission of the first image group Pg) tothe order reception server 5 by the communication device 12 of theclient computer 1 (step S111).

In a case where the data which represents the plurality of images and istransmitted from the communication device 12 of the client computer 1 isreceived (reception of the first image group Pg) in the communicationdevice 31 (one example of the first image group input device) of theorder reception server 5 (step S121), a consultation image group similarto the first image group Pg is detected from the order receptiondatabase 6 (step S122).

FIG. 22 represents a state of a plurality of consultation image groupsstored in the order reception database 6.

The order reception database 6 stores nine consultation image groups ofconsultation image groups Irg1 to Irg9 (less than nine consultationimage groups may be stored, or more than nine consultation image groupsmay be stored). The consultation image group may be a collection ofimages stored in a case where an order of the album is previouslyreceived from the user, or may be a collection of images in a case wherethe album is created using sample images.

The consultation image group Irg1 includes a plurality of images Ir 100to Ir 200. The album is created by extracting the image Ir 100 to theimage Ir 130 among the images Ir 100 to Ir 200. The image Ir 100 to theimage Ir 130 extracted from the consultation image group Irg1 will bereferred to as consultation images. Similarly, the consultation imagegroup Irg2 includes a plurality of images Ir 201 to Ir 300. The album iscreated by extracting the image Ir 201 to the image Ir 230 as theconsultation images among the images Ir 201 to Ir 300. The same appliesto other consultation image groups. In this example, the consultationimage group Irg1 is detected as the consultation image group similar tothe first image group Pg. Detection of the similar consultation imagegroup can be implemented using a method described in the first example.Images (not only the consultation images but also images included in theconsultation image groups) included in the plurality of consultationimage groups are distributed in a space corresponding to the featureamounts illustrated in FIG. 7 , FIG. 9 , FIG. 10 , and the like, and theimages included in the first image group Pg are distributed in acoordinate system. As a distance in the space between the image includedin the consultation image group and the image included in the firstimage group Pg is decreased, similarity between the image groups isincreased. A consultation image group for which the similarity isgreater than or equal to a threshold value is the consultation imagegroup similar to the first image group Pg. The threshold value may bepredetermined or may be changed until a consultation image group mostsimilar to the first image group Pg is obtained.

FIG. 23 illustrates each page of a consultation album Ab1.

In the consultation album Ab1, the consultation images Ir 100 to Ir 130are pasted in a template T1. The consultation album Ab1 has a first pageto a sixth page. Image pasting regions Ar 1 to Ar 5 are defined on thefirst page as illustrated in FIG. 24 . The consultation images Ir 100 toIr 104 are pasted in the image pasting regions Ar 1 to Ar 5.

With reference to FIG. 23 , the consultation images Ir 105 to Ir 109 arepasted on the second page, and the consultation images Ir 126 to Ir 130are pasted on the sixth page. The same applies to other pages.

In a case where the consultation image group Irg1 similar to the firstimage group Pg is detected, images (images for which similarity isgreater than or equal to a threshold value) similar to the consultationimages Ir 100 to Ir 130 of the detected consultation image group Irg1are extracted from the first image group Pg by the CPU 30 (one exampleof an extraction device) of the order reception server 5 (step S123).The threshold value may be predetermined or may be changed until a mostsimilar image is obtained.

FIG. 25 is one example of images included in the first image group Pg.

The first image group Pg includes 90 images of an image P11 to an imageP100.

FIG. 26 illustrates the consultation images Ir 100 to Ir 104 and theimages P11 to P15 similar to the consultation images Ir 100 to Ir 104.

As described above, the consultation images Ir 100 to Ir 104 areincluded in the consultation image group Irg1, and the images P11 to P15are included in the first image group Pg. The images P11, P12, P13, P14,and P15 are extracted from the first image group Pg as images similar tothe consultation images Ir 100, Ir 101, Ir 102, Ir 103, and Ir 104,respectively, that are pasted on the first page of the consultationalbum Ab1. The number of images extracted from the first image group Pgas the images similar to the consultation images may not be one and maybe plural in correspondence with one consultation image. In a case ofextracting one image from the first image group Pg in correspondencewith one consultation image, an image having the highest similarity withone consultation image may be extracted from the first image group Pg.However, extraction may not necessarily be performed in such a manner.One image may be decided in a case of pasting in the image pastingregion of the album.

Similarly, images are extracted from the first image group Pg for theconsultation images Ir 105 to Ir 130 pasted on a page of theconsultation album Ab1 other than the first page.

FIG. 27 illustrates a state where the images extracted from the firstimage group Pg are pasted in the consultation album Ab1.

For the first page of the consultation album Ab1, the images P11, P12,P13, P14, and P15 are extracted from the first image group Pg as imagessimilar to the consultation images Ir 100, Ir 101, Ir 102, Ir 103, andIr 104, respectively. The images P11, P12, P13, P14, and P15 are pastedby the CPU 30 (one example of an image pasting device) of the orderreception server 5 at respective positions of the image pasting regionsAr 1, Ar2, Ar3, Ar4, and Ar 5 in which the consultation images Ir 100,Ir 101, Ir 102, Ir 103, and Ir 104 are pasted.

FIG. 28 illustrates the first page of the album.

In FIG. 28 , the images P11 to P15 are pasted in the image pastingregions Ar 1 to Ar 5, respectively, in the same manner as theconsultation album Ab1 illustrated in FIG. 24 . As is perceived bycomparing FIG. 24 with FIG. 28 , an album having the same impression asan impression of the first page of the consultation album Ab1 isobtained.

Returning to FIG. 27 , the images P16, P17, P18, P19, and P20 areextracted from the first image group Pg as images similar to theconsultation images Ir 105, Ir 106, Ir 107, Ir 108, and Ir 109 that arepasted on the second page of the consultation album Ab1. The images P16,P17, P18, P19, and P20 are pasted at respective positions of imagepasting regions in which the consultation images Ir 105, Ir 106, Ir 107,Ir 108, and Ir 109 are pasted. The same processing is performed on otherpages. The images P26, P27, P28, P29, and P30 are extracted from thefirst image group Pg as images similar to the consultation images Ir126, Ir 127, Ir 128, Ir 129, and Ir 130 that are pasted on the lastsixth page of the consultation album Ab1. The images P36, P37, P38, P39,and P40 are pasted at respective positions of image pasting regions inwhich the consultation images Ir 126, Ir 127, Ir 128, Ir 129, and Ir 130are pasted.

The extracted images P11 to P40 are pasted in the same template T1 asthe template T1 in which the consultation images Ir 101 to Ir 130 arepasted (step S124 in FIG. 20 ). Particularly, since the images similarto the consultation images are pasted at the positions of the imagepasting regions in which the consultation images are pasted, the albumin which the extracted images P1 to P30 are pasted is an album havingthe same finish as the consultation album Ab1 in which the consultationimages Ir 101 to Ir 130 are pasted.

In a case where the album (electronic album) is generated, the albumdata representing the generated album is transmitted to the clientcomputer 1 from the order reception server 5 (step S56 in FIG. 21 ).

In a case where the album data is received in the client computer 1(step S42), the generated album is displayed (step S43). In a case wherethe album is corrected (step S44), the correction data is transmitted tothe order reception server 5 from the client computer 1 (step S45).

In a case where the correction data is received in the order receptionserver 5 (YES in step S57), the album data is corrected (step S58A).

In a case where the order button is pushed in the client computer 1 (YESin step S46), the order command is transmitted to the order receptionserver 5 from the client computer 1 (step S47).

In a case where the order command transmitted from the client computer 1is received in the order reception server 5 (YES in step S59), the orderreception processing is performed (step S60).

The same images as the consultation images extracted from theconsultation image group can be extracted from the first image group Pg,and the album having the same impression as the consultation album Ab1can be relatively simply created using the images included in the firstimage group Pg.

Modification Example

FIG. 29 is a part of a flowchart illustrating a processing procedure ofthe image processing system and corresponds to the processing procedurein FIG. 20 . In FIG. 29 , a part of the same processing as processingillustrated in FIG. 20 will be designated by the same reference sign.

As described above, in a case where the first image group Pg is inputinto the client computer 1 (step S131), the user designates a templateto be used in the album (step S132). For example, in a case where thefirst image group Pg is input by starting an album creation program, atemplate designation window (not illustrated) is displayed on thedisplay screen of the display device 11 of the client computer 1. Images(images showing contents of templates) of a plurality of templates aredisplayed in the template designation window, and an image of a desiredtemplate is selected by the mouse 17 (a template designation device).Then, the template is selected.

Next, the user designates a main person of the album (step S133).

FIG. 30 is one example of a main person designation window 140.

In a case where the first image group Pg is input into the clientcomputer 1, face detection processing is performed from the plurality ofimages included in the first image group Pg by the CPU 10 of the clientcomputer 1, and a face image that appears at a ratio of a certainfrequency or higher is detected. Detected face images F1 to F8 aredisplayed in the main person designation window 140. A decision button141 on which a text “decision” is displayed is formed at the lower rightof the main person designation window 140. Instead of the face imageappearing at a ratio of a certain frequency or higher, all face imagesdetected in the face detection processing may be displayed in the mainperson designation window 140. In a case where a face image desired tobe set as a main person by the user among the face images F1 to F8 isclicked by the mouse 17 and the decision button 141 is pushed, a personof the clicked face image is set as the main person. The number of mainpersons may be one or two or more.

The present examples are examples related to the album of the weddingceremony. Thus, as illustrated in FIG. 30 , a region R1 for designatinga face of the bridegroom and a region R2 for designating a face of thebride as the main person are disposed. In a case of specifying thebridegroom, the user drags and drops a face image to be specified as thebridegroom in the region R1 among the face images F1 to F8 (in a casewhere other face images are present, the other face images are displayedby scrolling). In a case of specifying the bride, the user drags anddrops a face image to be specified as the bride in the region R2 amongthe face images F1 to F8. Accordingly, the bridegroom and the bride canbe specified. Face images of persons estimated to be the bridegroom andthe bride may be displayed in advance by person recognition (of a personwho appears multiple times) or object recognition (of clothes and thelike normally worn by the bridegroom and the bride).

In the present example, a person specified as the bridegroom and aperson specified as the bride are distinguished as persons having rolesof the bridegroom and the bride. In the following description, in a caseof creating the album of the wedding ceremony, the term main person isused for a contrast between an example bridegroom and the bridegroom inthe first image group Pg and also a contrast between an example brideand the bride in the first image group Pg.

In a case where the main person is designated, the first image group Pginput into the client computer 1, identification data of the designatedtemplate, and data (face image data) representing the main person aretransmitted to the order reception server 5 from the client computer 1(step S134).

In a case where the first image group Pg, the identification data of thedesignated template, and the data (face image data) representing themain person transmitted from the client computer 1 are received in theorder reception server 5 (step S135), a consultation image group similarto the first image group is detected by the CPU 30 (one example of aconsultation image group detection device) of the order reception server5 from consultation image groups including the consultation imagespasted in the designated template among the consultation image groupsstored in the order reception database 6 (step S136). For example, evenin a case where the first image group Pg and the consultation imagegroup Irg1 are similar as described above (even in a case where thesimilarity is greater than or equal to the threshold value), theconsultation image group Irg1 is not extracted as the consultation imagegroup similar to the first image group Pg in a case where the designatedtemplate is not used in the consultation album Ab1 generated using theconsultation images included in the consultation image group Irg1.

In a case where the consultation image group is extracted, as describedabove, images similar to the consultation images included in theextracted consultation image groups are extracted from the first imagegroup Pg (step S123) (extraction processing of the similar images willbe described later), and the album is generated by pasting the extractedimages in the same template as the template (designated template) inwhich the consultation images are pasted (step S124).

FIG. 31 illustrates a relationship between a flow of wedding ceremonyand timings at which the consultation images Ir 100 to Ir 130 pasted oneach page of the consultation album Ab1 are captured at the weddingceremony.

For example, the wedding ceremony is started at time t0, and thebridegroom and the bride enter until time t1. The bridegroom and thebride are introduced from time t1 to time t2, and greeting of an honoredguest is performed from time t2 to time t3. A toast is made from time t3to time t4. A wedding cake is cut from time t4 to time t5. Conversationand dining are performed from time t5 to time t6. Guest entertainment isperformed from time t6 to time t7. Flower bouquets are presented toparents from time t7 to time t8. Acknowledgement of the bridegroom andthe bride is performed from time t8 to time t9. In an event such as thewedding ceremony, a general flow is determined, and a target captured ineach time range is also determined.

Consultation images of contents following the flow in the event arepasted in the consultation album. For example, consultation imagescorresponding to “entrance of the bridegroom and the bride” performedfrom time t0 to time t1 and consultation images (Ir 100 to Ir 104)corresponding to “introduction of the bridegroom and the bride” arepasted on the first page of the consultation album Ab1. Similarly,consultation images corresponding to “greeting of the honored guest”,consultation images (Ir 105 to Ir 109) corresponding to “toast”, andconsultation images corresponding to “wedding cake cutting” are pastedon the second page of the consultation album Ab1. Consultation imagescorresponding to “conversation and dining” are pasted on the third pageand the fourth page of the consultation album Ab1. Consultation imagescorresponding to “guest entertainment” are pasted on the fifth page ofthe consultation album. Consultation images corresponding to “flowerbouquet presentation” and consultation images (Ir 126 to Ir 130)corresponding to “acknowledgement” are pasted on the sixth page of theconsultation album Ab1.

A capturing time is recorded in a header of an image file. Thus, acapturing timing at which the image is captured is perceived byconsulting the capturing time. Among the images included in the firstimage group Pg, the similarity can be corrected to be increased for animage having the same capturing timing as a capturing timing of theconsultation image included in the consultation image group similar tothe first image group Pg, and the similarity can be corrected to bedecreased for an image having a different capturing timing as thecapturing timing of the consultation image included in the consultationimage group similar to the first image group Pg. An image having thehighest corrected similarity can be displayed in a selected imagedisplay region 90.

In a case where images similar to the consultation images among theimages included in the first image group Pg are simply extracted andpasted in the image pasting regions of the album, images notcorresponding to a scene may be pasted. For example, the consultationimages Ir 100 to Ir 104 illustrated in FIG. 21 are images on the firstpage and thus, are consultation images corresponding to a scene of“entrance of the bridegroom and the bride” from time t0 to time t1 and ascene of “introduction of the bridegroom and the bride” from time t1 totime t2. In this case, in a case where images that are similar to theconsultation images Ir 100 to Ir 104 but are captured at timings whichare not timings from time t0 to time t2, for example, timings from timet8 to time t9, are pasted on the first page, the generated album doesnot correspond to the flow of wedding ceremony. Thus, in this example,an image that is captured at a timing at which an image similar to theconsultation image is expected to be captured is pasted in the imagepasting region corresponding to the image pasting region in which theconsultation image is pasted.

As illustrated in FIG. 26 , even in a case where it is determined thatthe consultation image Ir 103 is similar to the image P4, the image P4is not pasted on the first page in a case where the image P4 is capturedat a timing from time t7 to time t8, because the consultation image Ir103 is pasted on the first page and images having the capturing timingfrom time t0 to time t2 are pasted on the first page. For example, theimage is pasted in any of the image pasting regions of the consultationimages Ir 126 to Ir 130 pasted on the sixth page illustrated in FIG. 27.

FIG. 32 is a flowchart illustrating a processing procedure of similaritycalculation processing (part of processing in step S123 in FIG. 29 ).

The similarity calculation processing illustrated in FIG. 32 isperformed on all images of the plurality of images included in the firstimage group. Similarity between the image included in the first imagegroup and the consultation image stored in the order reception database6 is calculated for all combinations of the images included in the firstimage group and the consultation images stored in the order receptiondatabase 6 (step S151).

In a case where an expected capturing timing of the consultation imagematches the capturing timing of the image included in the first imagegroup, it is determined that the image included in the first image groupis captured at the expected timing (YES in step S152), and the CPU 30(one example of a first similarity adjustment device) of the orderreception server 5 increases the similarity. The similarity may also beincreased in a case where an order of the consultation images pasted inthe consultation album matches a capturing order of the images.

Next, the CPU 30 (one example of a second similarity adjustment device)of the order reception server 5 also increases the similarity for animage having information similar to information about a face of theconsultation image (YES in step S154). The information about the face isinformation obtained from the face and is information about similarityof the face, information about the number of faces, or the like. Forexample, in a case where it is considered that the number of faces(faces considered as main subjects) included in the images is almost thesame, the CPU 30 of the order reception server 5 increases thesimilarity (step S155). In a case where the consultation album generatedusing the consultation images included in the consultation image groupis an album generated in the past by the user, the same album may begenerated later. In this case, a new album that is the same as the albumin the past can be generated by increasing the similarity of an imagesimilar to an image of the face included in the consultation imagepasted in the album in the past.

The number of appearances of a person is detected for the plurality ofimages included in the first image group Pg by the CPU 30. For an imagefor which the number of appearances is greater than or equal to athreshold value, or an image of a person designated as the main person(YES in step S156), the CPU 30 (one example of a third similarityadjustment device and a fourth similarity adjustment device) of theorder reception server 5 increases the similarity (step S157).

An image for which the adjusted similarity is greater than or equal tothe threshold value is set as the image similar to the consultationimage. Since the similarity is adjusted, a more appropriate album isgenerated.

In the above examples, images to be displayed are selected and displayedin accordance with a degree of similarity in a recommended image displayregion 101 and in accordance with a time series in a time-series imagelist display region 102. However, other selection methods and displaymethods may also be employed. For example, in the recommended imagedisplay region 101, an image to be displayed is selected in accordancewith the degree of similarity in a case where a degree of matching ofthe capturing timing is not considered. However, a display method suchas providing a mark indicating preference near an image for which thedegree of matching of the capturing timing is greater than a thresholdvalue may be employed. In addition, for example, while selection of animage to be displayed in accordance with the time series in thetime-series image list display region 102 is not changed from the aboveexamples, a display method such as providing a mark indicatingpreference near an image for which the similarity with an imagedisplayed in an example image display region 80 is greater than thethreshold value may be employed.

While the album of the wedding ceremony is created in the aboveexamples, an album other than the album of the wedding ceremony may becreated by consulting an example album. For example, an albumsummarizing school events for one year may be created. In this case,events performed for one year are generally performed in the same timeperiod each year. Thus, the similarity of images captured in the sametime period may be increased even in a case where a new album is createdusing an album created one year ago as an example.

While the bridegroom and the bride among the main persons aredistinguished as different roles in the above examples, a plurality ofmain persons may be handled without distinction depending on a creationpurpose of the album. That is, in a case where the number of mainpersons appearing in the example album is k (k is a natural number) andthe number of main persons in the first image group Pg is specified as m(m is a natural number), the similarity between an image in which any ofk persons is captured and an image in which any of m persons is capturedcan be increased by handling the images as images in which the mainpersons are captured, regardless of who is captured among k persons andwho is captured among m persons.

While image extraction processing and the like are mainly performed inthe order reception server 5 in the above examples, the above processingof the order reception server 5 may be performed in the client computer1 (image processing apparatus) in a case where the consultation imagegroups are stored in the client computer 1. In this case, it can beconfigured that a tablet terminal, a smartphone, a dedicated imageprocessing apparatus, or the like is used instead of the client computer1.

While the album is generated using the images extracted from the firstimage group in the above examples, the invention is not limited to acase of generating the album and can also be used in a case where ashuffle print in which a plurality of images are pasted on an imageproduct such as one paper sheet is made, a case where an image isextracted from a plurality of images and the extracted image is printedone sheet at a time, and the like.

Processing units executing the above processing include not only the CPU10 of the client computer 1 and the CPU 30 of the order reception server5 functioning as various processing units by executing software, butalso a programmable logic device such as a field-programmable gate array(FPGA) having a circuit configuration changeable after manufacturing, adedicated electric circuit such as an application specific integratedcircuit (ASIC) that is a processor having a circuit configurationdedicatedly designed to execute a specific type of processing, and thelike.

One processing unit may be configured with one of various processors ormay be configured with a combination of two or more processors of thesame type or different types (for example, a plurality of FPGAs or acombination of a CPU and an FPGA). As an example of configuring aplurality of processing units with one processor, first, a form inwhich, as represented by a computer such as a client computer or aserver, one processor is configured with a combination of one or moreCPUs and software and this processor functions as a plurality ofprocessing units is available. Second, a form in which, as representedby a system on chip or the like, a processor that implements a functionof the entire system including a plurality of processing units in oneintegrated circuit (IC) chip is used is available. Accordingly, variousprocessing units are configured using one or more of various processorsas a hardware structure.

Furthermore, the hardware structure of various processors is morespecifically an electric circuit in which circuit elements such assemiconductor elements are combined.

What is claimed is:
 1. An image processing apparatus comprising: a firstimage group reception device that receives an input of a first imagegroup; and an image extraction device that, among a plurality ofreference image groups each having information about an extracted partof images, extracts, from the first image group, an image similar to thepart of the images extracted from a second image group similar to thefirst image group.
 2. An image processing apparatus comprising: a firstimage group input device that receives an input of a first image group;and an extraction device that extracts, from the first image group, aplurality of images for which similarity with a plurality ofconsultation images extracted from a consultation image group is greaterthan or equal to a threshold value.
 3. The image processing apparatusaccording to claim 2, wherein a plurality of the consultation imagegroups are present, the image processing apparatus further comprises aconsultation image group detection device that detects, from theplurality of consultation image groups, the consultation image group forwhich similarity with the first image group is greater than or equal toa threshold value, and the extraction device extracts, from the firstimage group, a plurality of images for which similarity with theplurality of consultation images extracted from the consultation imagegroup detected by the consultation image group detection device isgreater than or equal to a threshold value.
 4. The image processingapparatus according to claim 2, wherein each consultation image of theplurality of consultation images extracted from the consultation imagegroup is pasted in an image pasting region of a template, and the imageprocessing apparatus further comprises an image pasting device thatpastes, in the image pasting region of the template, an image for whichsimilarity with the consultation image pasted in the image pastingregion of the template is greater than or equal to a threshold valueamong the plurality of images extracted in the extraction device.
 5. Theimage processing apparatus according to claim 4, wherein the imagepasting device pastes, in an image pasting region corresponding to theimage pasting region in which the consultation image is pasted, theimage for which the similarity with the consultation image pasted in theimage pasting region of the template is greater than or equal to thethreshold value among the plurality of images extracted in theextraction device.
 6. The image processing apparatus according to claim4, wherein the image pasting device pastes, in an image pasting regioncorresponding to the image pasting region in which the consultationimage is pasted, an image for which the similarity with the consultationimage pasted in the image pasting region of the template is greater thanor equal to the threshold value and that is captured at a timing atwhich an image corresponding to the consultation image is expected to becaptured.
 7. The image processing apparatus according to claim 4,further comprising: a template designation device that designates onetemplate from a plurality of templates, wherein each consultation imageof the plurality of consultation images extracted from the consultationimage group is pasted in the image pasting region of the templatedesignated by the template designation device.
 8. The image processingapparatus according to claim 2, wherein an order of the plurality ofconsultation images is determined, and the image processing apparatusfurther comprises a first similarity adjustment device that increasesthe similarity with the consultation image for an image captured in anorder corresponding to the order of each consultation image of theplurality of consultation images or an image captured at a timing atwhich an image corresponding to the consultation image is expected to becaptured among images included in the first image group.
 9. The imageprocessing apparatus according to claim 2, further comprising: a secondsimilarity adjustment device that increases the similarity with theconsultation image for an image having information similar toinformation about a face included in the consultation image among imagesincluded in the first image group.
 10. The image processing apparatusaccording to claim 2, further comprising: a third similarity adjustmentdevice that increases the similarity with the consultation image for animage including a person for which the number of appearances in theimage is greater than or equal to a threshold value among imagesincluded in the first image group.
 11. The image processing apparatusaccording to claim 2, further comprising: a person designation devicethat designates a desired person among persons appearing in imagesincluded in the first image group; and a fourth similarity adjustmentdevice that increases the similarity with the consultation image for animage including the person designated by the person designation device.12. The image processing apparatus according to claim 11, furthercomprising: a face image detection device that detects a face image fromthe images included in the first image group; and a face image displaycontrol device that controls a display device to display the face imagedetected by the face image detection device on a display screen, whereinthe person designation device designates the desired person bydesignating the face image displayed on the display screen.
 13. Theimage processing apparatus according to claim 2, further comprising: animage product creation device that creates an image product using theimages extracted by the extraction device.
 14. An image processingmethod comprising: receiving, by a first image group reception device,an input of a first image group; and among a plurality of referenceimage groups each having information about an extracted part of images,extracting, by an image extraction device, from the first image group,an image similar to the part of the images extracted from a second imagegroup similar to the first image group.
 15. An image processing methodcomprising: receiving, by a first image group input device, an input ofa first image group; and extracting, by an extraction device, from thefirst image group, a plurality of images for which similarity with aplurality of consultation images extracted from a consultation imagegroup is greater than or equal to a threshold value.
 16. A recordingmedium storing the computer-readable program controlling a computer ofan image processing apparatus, the program controlling the computer ofthe image processing apparatus to perform: receiving an input of a firstimage group; and among a plurality of reference image groups each havinginformation about an extracted part of images, extracting, from thefirst image group, an image similar to the part of the images extractedfrom a second image group similar to the first image group.
 17. Arecording medium storing the computer-readable program controlling acomputer of an image processing apparatus, the program controlling thecomputer of the image processing apparatus to perform: receiving aninput of a first image group; and extracting, from the first imagegroup, a plurality of images for which similarity with a plurality ofconsultation images extracted from a consultation image group is greaterthan or equal to a threshold value.